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ABSTRACT 



This paper describes the design of a Web-based learning 
environment for leadership facilitators in a United States military 
organization. The overall aim of this project was to design a prototype of an 
online learning environment that supports leadership facilitators’ knowledge 
development in the content area of motivation. The learning environment was 
designed to provide new learning opportunities for facilitators to build 
content knowledge through interactive learning activities and experiences 
with other members of their own community. This project was managed as a 
concurrent solution design process incorporating elements of strategic 
thinking, change creation, work environment design, and rapid prototyping. 

The front-end analysis and impact evaluation were built into the design 
process. The use of the iterative solution design process (with the iterative 
design phases of performance analysis, prototype design, and prototype 
re-design) allowed for increased efficiency and effectiveness in making 
ongoing refinements and improvements to the prototype. The project team 
agreed upon three broad areas of learning environment design- -analyzing, 
designing, and continuously improving- -and these principles were used as the 
foundation for a pre-prototype evaluation plan. The plan included questions 
derived from a variety of performance improvement theories and models. A 
solution design decision aid assisted in evaluating the effectiveness of the 
learning and performance environment. Specific strategies and recommendations 
related to utilization and ongoing support of the prototype that was designed 
and delivered to the client were proposed. These focused in two key areas: 
adopting and diffusing the product within the target organization; and 
continuously improving the design and usefulness of the current prototype 
once it has been adopted by the target organization. Critical diffusion and 
work environment factors are identified, with recommendations, to highlight 
some of the biggest challenges to project success. Brief descriptions of each 
major component of the Web-mediated learning and performance environment are 
then offered, with recommendations for continuous improvement and maintenance 
of each on the Web site. ( AEF) 
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Introduction 

This paper describes the design of a web-based learning environment for leadership facilitators in 
a U.S. military organization. The overall aim of this project was to design a prototype of an online learning 
environment that supports leadership facilitators’ knowledge development in the content area of 
motivation. This learning environment was designed to provide new learning opportunities for facilitators 
to build content knowledge through interactive learning activities and experiences with other members of 
their own community. In order for this learning and performance environment to continually improve and 
be effective in the long run, issues concerning diffusion of the technology and ongoing learning community 
development should be addressed. 

The iterative design phases used to develop this web-based application are illustrated in Figure 1 
below. The processes of analysis, design, and evaluation are often thought of as unique, stand-alone 
events. This project was managed as a concurrent solution design process incorporating elements of 
strategic thinking, change creation, work environment design, and rapid prototyping. The front-end analysis 
and impact evaluation were built into the design process. 
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Figure 1. Iterative Design Phases 



The use of this iterative solution design process allowed for increased efficiency and effectiveness 
in making ongoing refinements and improvements to the prototype. The designers utilized the learning 
environment design principles identified in the next section to drive the major tasks and to assure project 
quality. 

Additionally, actions considered necessary for ongoing support of the web-based environment 
were summarized in two major areas: 1) Diffusion of the technology into the facilitator community, and 2) 
Continuous improvement and adaptation of the site by the community. 



Key learning environment design principles and concepts 

The project team agreed upon three broad areas of learning environment design. These 
components address a wide array of solution analysis, design, development, implementation, evaluation 
and diffusion issues. For the sake of brevity, these principles have been listed, but not fully elaborated 
upon, below. These principles are useful as guides to match project needs with learning environment 
features. 
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Analyzing 

• Identify people who are accountable for the results of learners and engage them in the design goal 
setting process. 

• Identify components and levels of the organizational system where performance-based results are 
dependent on learning achievement. 

• Determine measures of success of learning and performance on an individual and organizational 
basis. 

• Plan how you will evaluate this success (pre-post learning). 

• Identify non-training, work environment factors that could contribute to the success or failure of 
the web-based learning environment. 

• Create a plan for diffusion and adaptation of the learning environment within the organization. 

• Create an awareness and knowledge of the analysis process. 

Designing 

• Create learning objectives from performance objectives, based on task, job and/or performance 
analyses. 

• Create activities that gain the learners’ attention by engaging them in the learning process. 

• Create activities that show learners how the instruction will be of benefit to them. 

• Provide ways for the learners to practice learning activities and get feedback on their performance. 

• Create ways for learners to assess their own performance in a meaningful way that simulates the 
actual performance as close as possible. 

• Create ways for learners to access a common knowledge base of resources on a given topic. 

• Provide ways for learners to interact with instructors and with other learners. 

• Provide ways for learners to gain multiple perspectives on a particular problem solving approach. 

• Provide a means for learners to access knowledge based on previous learners’ experiences. 

• Engage members of the learning community or organization in adaptation of the learning 
environment for better fit within the current culture. 



Continuously Improving 

• Collect data to measure the success of learners and specific learning environment design 
processes. 

• Identify non-training, work environment factors that could contribute to the success or failure of 
the web-based learning environment. 

• Adapt learning design processes / adapt environment to optimize performance of learners. 

• Collect data to measure the success of performers and the organization in meeting goals as related 
to learning activities and objectives. 



These key design principles were used as the foundation for a pre-prototype evaluation plan. The 
plan included questions derived from a variety of performance improvement theories and models. A 
solution design decision aid assisted in evaluating the effectiveness of the learning and performance 
environment. Tables 1 through 5 are excerpts of this decision aid. 
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Table 1: Strategic Organizational Context (adapted from Ely; Kaufman; Rogers) Evaluation 
Probes 

Strategic Linkages 

■ How are project objectives linked to the job, work center, and organizational missions and vision? 

■ Has a continuous improvement plan including impact evaluation been completed for this project? 

Innovation-Change Adoption 

■ Is there a primary stakeholder, and owner of this project that will support its adoption and diffusion? 

■ What are barriers that may prevent long-term or continued success of the project? 



Table 2: Organization-Work Center Context (adapted from Gilbert; Wedman & Graham) 
Evaluation Probes 

Expectations and Feedback 

■ Do performers know what is expected of them on the job? Do they know when they have done the job 

correctly? 

Tools and Information 

■ Will supporting documentation, job aids, and other performance support be available? 

■ Is there time in the work schedule for performers to use new skills learned in training? (TT) 

Rewards and Incentives 

■ Is completion of tasks rewarded or punished? 



Table 3: Performer Context (adapted from Keller; Wedman & Graham) Evaluation Probes 

Motivation and Self- Concept 

■ Do workers want to do good work? 

■ Do they monitor their own performance? 

Performance Capacity 

■ Are workers physically able to complete job tasks? 

■ What degree of flexibility with respect to work pace, structure, and organization is 

required of workers? 

Skills and Knowledge 

■ Do workers have adequate knowledge and skills to do the job well? 

■ Which skills taught in training are not being used on the job? Why aren’t they being 

used? (TT) 



Table 4: Instructional Design (adapted from Dick and Carey; Keller) Evaluation Probes 

Practice/Assessment: 

■ Does practice seem relevant to on-the-job expectations and performances? 

■ Do practice items/assessments match objectives? 

Feedback: 

■ Does feedback promote learner satisfaction with the learning experience? 

■ Does feedback encourage further exploration? How could exploration be further 
encouraged? 

Interactivity: 

" If appropriate to objectives, is interaction among learners supported? 



12 



Table 5: User-Computer Interface (adapted from Nielsen) Evaluation Probes 

Visibility of system status 

" Does the learner know where he/she can/should go next? 

Match between system and the real world 

" Does the online learning environment represent or support the work environment? 

Consistency and standards 

" Is there a standard convention used to identify key areas of the site? 

Aesthetic and minimalist 

■ Is information presented progressively from more general to more specific, thus allowing 
learners to drill down to in-depth information as necessary? 



To accomplish the objectives of this particular project, the iterative design phases (performance 
analysis, prototype design, and prototype re-design) were integrated with the design principles to produce a 
functional prototype. A review of this integration process and recommendations for implementation of the 
prototype follow. 



Performance Analysis 

Performance analysis was a process of defining the results expected by the leadership community 
and the facilitator practices that could lead to achievement of those results. Current and desired facilitator 
results, practices, and attitudes were analyzed during this phase. Measurable results, critical performance 
and learning measures, achievable facilitator practices, and work environment factors and conditions that 
support or hinder performance were profiled. Prototype design began after performance objectives and the 
flow of information in the leadership facilitator community were identified. The results of analysis were 
used to guide the planning for the evaluation, design and implementation of the learning and performance 
environment solution. 



Prototype Design and Re-design (Continuous Improvement) 

Participatory design processes are effective in building stakeholder and user commitment to the 
success of application systems through their active participation in the analysis of requirements and the 
specification of the system design. For this project, each design iteration and improvement was based on 
discussions with key sponsors, stakeholders and potential users. Participants and designers collaborated to 
develop a potentially valuable product, identify possible product improvements, and generally provide one 
another with guidance in the ongoing design of the product. 



During this concurrent design and evaluation process, interviews and focus groups were conducted 
during which visual prototypes clarified concepts that were being discussed. Initial prototypes were 
PowerPoint si ides, which were transformed into a prototype web site using simple WYSIWYG web design 
tools. To assist in this transformation, members of the target audience provided initial feedback regarding 
the site look and feel, and overall concept. Following initial feedback, a “joint application design” meeting 
was held with sponsors (designers, project managers, directors) during which the first relatively primitive 
web-based site was reviewed live. The purpose of this meeting was to collect data to assist in design 
decision-making. This review followed a somewhat orderly process with a set of structured interview 
questions accompanying the review of each screen. A major goal of this design iteration was to ensure that 
the organizational learning and performance objectives of the site were addressed. 



Following this meeting, revisions were made accordingly in order to prepare the site for usability 
testing / evaluation. The evaluation process included, but was not limited to, assessment of the aesthetic 
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quality of the interface, and following redesign, assessment of the content, learning activities, and 
achievement of objectives. 



The first level of evaluation was with facilitators at a large training unit with more than 100 
facilitators. Upon familiarizing the facilitators with the basic goals of the site, they were provided the URL 
and asked to provide feedback (we have been requested to not release the results of these evaluation 
processes by the client). 



Following this process, the evaluation processes listed below were also completed: 

• Learner/Performer Evaluation - A survey was placed on the website to gather input from 
facilitators regarding their reactions to site content and learning activities, and suggestions for 
improvement. Users also had the opportunity to provide feedback on the site discussion 
board. 

• Individual interviews with novice users were conducted to observe a typical user interaction 
and resolve any barriers to navigation and learning. The user went through each activity and 
made suggestions for the content and layout of the site. 



Specific strategies and recommendations related to utilization and ongoing support of the 
prototype that was designed and delivered to the client were proposed. These strategies and 
recommendations were focused in two key areas: 1 ) Adopting and diffusing the product within the target 
organization; and 2) Continuously improving the design and usefulness of the current prototype once it has 
been adopted by the target organization. 



Adopting and Diffusing the Product: Implementation Strategies 

The most well designed products often go unused or unappreciated within organizations often due 
to the lack of sponsorship and inadequate attention to the diffusion of the new technology. During the 
performance analysis phase of this project, many non-training, work environment factors that could 
contribute to the success or failure of the web-based learning environment were identified. Critical 
diffusion and work environment factors are identified below to highlight some of the biggest challenges to 
project success with this particular target audience. Recommendations relative to these factors have also 
been included. 



Redefining/Restructuring, and Clarifying Relationships/Roles 

These two critical stages in the diffusion of a technology within an organization have been 
documented by researchers in many different organizational settings (Rogers, 1 995). Following initial 
design, development and evaluation of the web-based learning environment, it was expected that members 
of the facilitator community will begin the process of redefining/restructuring and clarifying roles related to 
this technology. Also see Communities of Practice by Wenger ( 1 998). 



The following recommendations were put forth in an effort to assist in the initial adoption, and 
redefining/clarifying of the prototype. Long-term evaluation of the site is possible only after adoption and 
utilization of the prototype by members of the facilitator community. 

Recommendations: 

• Identify decision-makers that will advocate the site. This is the identification of key 
stakeholders within the facilitator community who take responsibility for the product. The project 
sponsor and individual training units will be responsible for identifying key adopters, change agents, 
and problem-solvers who will champion the use of this innovation by members of the facilitator 
community. These role players will be active participants in the design, re-design, and evaluation of 
the application. 
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• Identify implementation managers who will diffuse the site throughout the facilitator 
community. This is a critical aspect of the early design and development of any new product. These 
are early adopters that must assume the burden of “selling” the product to others, as well as taking on 
much of the responsibility for its success. One strategy that stakeholders may use is to integrate this 
tool with other performance support tools currently in use. 

• Introduce the site during ongoing activities already existing in the environment. 

- Introduce the site during initial mentoring, 

- Have trainers and mentors introduce the site as a tool for individual and job 
development The site could help experienced facilitators mentor and coach less experienced 
facilitators. Facilitators may be positively influenced to use the site if it is encouraged by their 
mentors as a useful way to develop skills and build new knowledge. 

- Introduce the site as part of facilitator training. Introducing the site as a tool that 
can be used by facilitators to build knowledge and skills related to job duties and responsibilities 
will increase its potential impact and overall effectiveness. 

Resources for ongoing design and development 

Identification of adequate resources, human and financial, to support the maintenance and 
continued growth of the application is crucial to success. This is an often-neglected aspect of computer- 
based application development, especially if the technology requires frequent updating. Web-based 
technologies are easier to update than other technologies and thus represent a potential long term cost 
savings. 

Recommendations: 

• Form a team responsible for learning and redefining the website. Although time and 
monetary requirements are minimal, it will be essential to identify key personnel who will be 
responsible for supporting ongoing maintenance of the site. It was suggested that internal facilitators 
who are familiar with the facilitator community and current practices be trained in change and redesign 
processes. A technical support resource responsible for updating and refining software and hardware 
will also be required for continual improvement of the site. 

• Conduct prototype review groups with potential users and opinion leaders. Continual 
feedback from the learning community is helpful in redefining and redesigning the site to support and 
build on current course content. 

Incentives 

Incentives and reward systems are one of the least understood yet most effective methods of 
improving performance in the workplace. The benefits of participating in the learning environment include 
improved performance, self-development, and professional growth. 

Recommendations: 

• Official “release” time for participation in learning activities, and recognition for 
participation are highly recommended. The costs of release time can be considerable but the potential 
benefits of participation, in terms of improved performance, are well worth the cost. 

• Facilitators should receive positive consequences and encouragement from their 
command to use the site as part of their ongoing skill development. 

Process Management 

Technical and human resource management will be required to ensure consistent, supportive, and 
performance-oriented system implementation. The interrelationship of this new process with other 
facilitator processes is part of the role of project leaders. 

Recommendations: 

• Leaders of this project within the facilitator community must be fully engaged in the 
process of creating, refining, communicating and using knowledge that is supported by technology. 
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• Periodic feedback should be elicited from facilitators to continually improve and add 
areas of interest to facilitators that are actually using the site. 



Equipment (see Site Technical Maintenance section) 

Hardware and software, physical information infrastructure, and documentation will be required to 
support initial and ongoing system development. 



Recommendations: 

• Current computer hardware and software must meet minimum specifications in order to 
capitalize on emerging technologies. This is probably the most expensive (monetarily) of all 
implementation factors, but can pay off in learning and performance effectiveness in the long term. 



Continuous Improvement of the Learning and Performance Environment 

The following section lists the major components of the web-mediated learning and performance 
environment. These components address the learning objectives required to achieve selected facilitator 
performance requirements or best practices. By addressing the best practices that were identified through 
the performance analysis, this learning environment has a greater likelihood of having a positive impact on 
facilitator, performer, and organization readiness. The pages that follow offer a brief description of each 
component of the learning and performance environment, and recommendations for continuous 
improvement and maintenance of each component on the website. Many of the recommendations included 
here were generated during the participatory design processes. 



Learning Environment Component: Practice 

Purpose : To enhance content knowledge and problem solving ability in the area of motivation 
through a variety of Self-Assessment/ Content Mastery activities. 



These activities include multiple choice, matching and “essay’Vcase study response formats in an 
effort to enhance situational problem- solving ability. 



Recommendations for Ongoing Development: 

- Monitor and update self-assessment items to reflect current course content. 

- Continually develop new items that challenge users and reflect current research and practice 
in the area of motivation. 

- Ensure relevant, corrective feedback for each item. 

- Add tutorials that provide a review of classroom material for each level of the course guides 
for motivation. 

- Identify current problems to which facilitators can apply basic problem solving skills to solve 
larger issues/concerns. 

- Monitor case study submissions to ensure learning opportunities go beyond that presented in 
the classroom, 

- Continually add new case studies. ’Expert' facilitators could be identified to update and 
maintain relevant case studies. 

- Review submissions by peers and revise as necessary. 

- Revise case studies as field experiences change or as current case studies become outdated. 

- Hold expert forums in the chat room and / or archive interview transcripts. 
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Learning Environment Component: Discussion Board 

Purpose : To provide current learners with a platform to build usable knowledge for current and 
future facilitators and leaders through interaction. 

Features include: Message Board with chat feature to share ideas, experiences, discussions, etc., 
and Synchronous and Asynchronous communication options. 

Recommendations for Ongoing Development: 

Cleaning up/Monitoring 

- Continually monitor information for accuracy. 

- Delete outdated or incorrect information. 

- Review other discussion boards that may better suit user requirements (see Site Technical 
Maintenance section). 

Maintaining Interactions 

- Periodically post new discussion topics. 

- Encourage participation by “graduates” now on the job. 

- Provide positive consequences for facilitators to interact with others on the discussion 
board. 

- Reward expert facilitators or SME's who participate in scheduled discussions. 



Learning Environment Component: Help 

Purpose : To provide assistance and reduce frustration in navigating through the site. Features 
include a Site Map, Technical Help with FAQ’s and site tips. 



Recommendations for Ongoing Development: 

Updating/Revising 

As facilitators become more familiar with the site, Help section will require frequent 
review/revision. 

Monitor and delete/update FAQ’s. Facilitators should be encouraged to ask questions of 
their peers and to respond to questions appropriately. 

Revise site map/navigational tips based on revisions and advancements to the site. 



Learning Environment Component: Knowledge Base 

Purpose : Where all other features converge to allow for storage of objective (known facts, 
theories, procedures) knowledge and to promote constructed (new) knowledge. 



Features include: Library with motivation-related readings, instructional resources, and links to 
other organization web sites. 



Recommendations for Ongoing Development: 

Monitor and update/ delete “dead links ” to remain current. Many online resources are deleted by 
the site authors after a certain period of time and therefore many links may become "dead” links. 

Links should reflect current course content and should not present views conflicting with current 
curriculum. 

Expert contributions should be archived and continuously updated. 

Archived models/theories should be created and easily accessed. 

A drop-down menu can be created so that reactions to information found in web links can be 
accessed under each link in the Library. 
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Learning Environment Component: Evaluation 

Purpose : To provide an opportunity for the user to aid in the continual development of the site. 

Users complete a profile that identifies them by location and level experience, among other 
factors, and solicits their feedback on the site. This profile is also suitable as a Level One (Kirkpatrick) 
reaction evaluation tool. 

Recommendations for Ongoing Development: 

Evaluation responses should be logged and continuously monitored to assist in 
usefulness of data in continuous site improvement . 

Survey questions (as well as the profile) should be reviewed based on changes to the site 
and evaluation data expected . 

As updates are made to the site, face-to-face evaluations should be conducted in order to 
observe interactions and to receive feedback from facilitators regarding ease of navigation and 
usefulness of activities. 



Site Technical Maintenance 

The following is a brief description of the tools and applications used in the development of this 
project. The designers provided sufficient information for an Information Technologist to understand the 
basic architecture and technical features of this product for the purpose of continual technical development. 
Where applicable, possible alternative applications to the ones used in the prototype design were suggested. 



Platform : Windows NT 4 running on Novell Network Server. 

■ Maintenance Suggestions : All site HTML and graphic files should be transferred to a Navy server 
for ease and control of ongoing site maintenance. 

Web Interface : The site is optimized to run on MS Internet Explorer and/or Netscape Navigator 
browsers. 

Development Applications : 

Microsoft FrontPage 2000™ 

Macromedia Fireworks 3™ 

Hypertext Markup Language (HTML) 

Macromedia Flash 4™ 

Notepad 

Microsoft Access 2000™ 

Adobe Photoshop 5.5™ 

Visual Basic Scripting™ 

Web Crossing™ 

There are various online course development software applications that have also gained 
popularity. WebCT, Blackboard Courselnfo, Phoenix Pathlore, and Construe are just a few examples. 
Alternatively, this site was created with a variety of COTS products that allowed design flexibility for 
ultimate user customization. 
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